-
-
Notifications
You must be signed in to change notification settings - Fork 258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce pex3 venv create
.
#2140
Conversation
I know this PR is in a draft but I just wanted to say that instead of a |
Yeah, I'm not super happy about a The thing is, the code used is >50% the same and the use cases I'm aware of are similar; so I've smooshed them together. The motivating use case for Saying that out loud, perhaps |
This new subcommand can create either a venv or just populate a flat `sys.path` directory entry (ala `pip install --target`) given a set of requirements to resolve, potentially from a lock or an existing PEX file, but otherwise from indexes and find links repos. Unlike the sibling `venv` `pex-tool` subcommand, the target can be selected and, in the flat `sys.path` directory entry case, it can be a foreign platform. Fixes pex-tool#1752 Fixes pex-tool#2110
Alright, reviewers thanks in advance - this is large. It's mainly just the new sub-command and its integration tests, but there was some code moved around and a small bit of new code in Creating a venv directly from a lock (subset) was what I needed for a-scie/lift, but it was easy enough to roll in features satisfying a few Pex issues as well as the unrealized goal @benjyw had over in Pants of having the lambda function package results be exactly what AWS / GCP / Azure call for in both deployed zip and, in the case of AWS, Lambda layers. |
from typing import DefaultDict, Iterable, Iterator, List, Optional, Tuple, Union | ||
|
||
|
||
def find_dist( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This block of code to ensure pip is installed moved from the existing venv
PEX_TOOL here: https://github.com/pantsbuild/pex/pull/2140/files#diff-44e8fd52cd3a63a2d3f2fc516418c2aa55e8ee96f6eb82b2186765a8b853922fL83-L180
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I created a pex from this commit and it seems to work fine for me. I tested both venv and flat layouts from a lockfile with a local interpreter.
I don't quite understand the --force
flag here with the venv layout. If I provide --force
and the directory exists, is the venv destroyed and recreated? I my understanding is that the pex-tools
command deletes the venv and recreates it entirely.
Correct.
That is not correct when doing partial exports ( These are semantics I'm not convinced about, although a hard line is supporting the srcs / deps split. If you have alternate semantics that make more sense and support the split let me know. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice!
I experimented with this to create lambda packages (including layers), and it worked well, using --complete-platform=...
and --layout=flat-zipped
along with either --scope=deps --prefix=python
for a layer or --scope=srcs
for the main package.
@@ -549,6 +549,7 @@ def _populate_sources( | |||
dst=dst, | |||
exclude=( | |||
"__main__.py", | |||
"__pex__", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A venv PEX never needed the magic __pex__
import hook, which won't work anyhow since a venv PEX has no Pex .bootstrap code carrying the import hook implementation code.
This new sub-command can create either a venv or just populate a flat
sys.path
directory entry (alapip install --target
) given a set ofrequirements to resolve, potentially from a lock or an existing PEX
file, but otherwise from indexes and find links repos. Unlike the
sibling
venv
pex-tool
sub-command, the target can be selected and,in the flat
sys.path
directory entry case, it can be a foreignplatform.
Fixes #1752
Fixes #2110
Fixes #2111